RISEkbmRasch

an R package to simplify and integrate analysis and documentation with Quarto

Magnus Johansson, PhD

RISE Research Institutes of Sweden

2023-06-15

Agenda

  • a brief background of the package
    • reproducibility, transparency & quality
  • walkthrough of Quarto/R code and its output
    • using the package vignette as an example (available online)

Useful links - github.com/pgmj

Why R?

  • open source & free
  • potentially easier for others to reproduce and learn from your results
  • available for Windows/Mac/Linux/etc

But…

Plots and other output still needs copying to a text document to collect the output and document the analysis

  • challenging to reproduce results
    • especially if you go back to an old project

Quarto

https://quarto.org

  • documentation in the same document as the analysis code!
  • can output PDF, Word, HTML documents (and more)
  • makes reproducibility simple
  • (yes, it is like Rmarkdown)
  • pre-installed with Rstudio

Brief history

  • I tried ordering RUMM but didn’t get a response
  • Took a course on IRT/CAT with R
  • Created code to recreate/develop output of Winsteps/RUMM
  • Workshop with colleagues - “Maybe make functions to simplify?”

Before & after functions

Before

After

RIpcmPCA(df)

What is an R package?

  • a collection of functions()
  • RISEkbmRasch relies 100% on other packages
    • it can be described as a “wrapper” package
    • it is also an “opinionated” package

Package ambitions

  • make it as simple as possible to get key tables and figures
  • you can choose cut-off values for highlighting in most functions, for instance:
    • item fit over/under a certain value
    • residual correlations relative to average residual correlations
  • more flexibility gradually added (but also adds complexity)

Notes on choices

There are multiple R packages for Rasch analysis.

  • We went with eRm primarily
    • handles dichotomous and polytomous data
    • uses CML, conditional maximum likelihood
      • “specific objectivity”
      • ordinal sum score as a “sufficient metric”
  • mirt for Yen’s Q3 residuals
  • psychotree for DIF (differential item functioning)

Simulation study coming

Partial credit model (PCM) analysis

  • eRm with CML
  • TAM with MML

Comparisons with variation in sample size and targeting. May also produce a reasonable basis for assessing power for Rasch analysis. R code will be included.

What’s in the package?

  • Descriptive analysis
    • distribution of data
    • missing data
    • Guttman “heatmap”

Required data structure

  • one dataframe with item data ONLY
    • coded as integers starting with 0 for lowest response category
  • one dataframe with item descriptions
  • (DIF variables as separate vectors)

Benefits

  • you can make a template analysis file
    • makes it harder to miss important steps
      • quality assurance
  • easier for others to understand your analysis process, step by step
    • easier for yourself to go back to old analyses…
  • transparency in decision making
  • complete reproducibility if data is shared

Report everything!?

  • You can share a fully documented report file as an appendix document with the preprint
  • Example:

Rozental, A., Forsström, D., & Johansson, M. (2023). A Psychometric Evaluation of the Swedish Translation of the Perceived Stress Scale: A Rasch Analysis [Preprint]. In Review. https://doi.org/10.21203/rs.3.rs-2699284/v1

A note on templates

Our group at RISE have made an analysis template based on our preprint, in which we propose a reporting standard for psychometric analyses. It builds on Tennant & Conaghan’s 2007 paper and others.

Johansson, M., Preuter, M., Karlsson, S., Möllerberg, M.-L., Svensson, H., & Melin, J. (2023). Valid and Reliable? Basic and Expanded Recommendations for Psychometric Reporting and Quality Assessment. OSF Preprints. https://doi.org/10.31219/osf.io/3htzc

magnus.p.johansson@ri.se